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Summary 

Cost  is  one  of  the  key  performance  parameters  of  the  defense  acquisition  system;  cost  is 
controlled  and  traded  against  schedule  and  technical  performance  of  weapon  systems.  A  new 
measure  of  cost  growth  (and  thus  cost  control  and  tradeoffs)  was  developed  to  help  look  for 
longitudinal  performance  trends  over  the  last  three  decades.  This  measure  examines  annual  cost 
changes  across  major  contracts  for  development  and  early  production  on  major  programs.  This 
should  not  be  mistaken  for  the  total  costs  of  these  programs  because  it  excludes  non-contracted 
costs  and  the  majority  of  production  contracts.  Nonetheless,  we  expect  that  the  cost- 
reimbursement  and  fixed-price  incentive  contracts  that  this  study  examines  should  show  the 
same  trends  as  other  measures  of  total  program  cost  growth — in  part  because  production 
contracts  have  much  lower  cost  growth  than  these  development  and  early  production  contracts. 
Such  changes  reflect  added  work  and  overruns  after  adjusting  for  inflation. 

A  5-year  moving  average  of  the  annual  changes  was  used  to  smooth  the  data,  revealing  the 
patterns  visible  in  Figure  1.  Statistical  analysis  found  that  the  combined  effect  of  three  types  of 
factors — plus  a  constant  base  level  of  growth — accurately  models  growth  on  these  contract 
(Figure  2  shows  how  closely  the  model  fits  the  actual  data). 

First,  the  pattern  of  growth  partially  follows  the  defense  budget.  In  other  words,  there  is  strong 
evidence  that  changes  in  budget  have  an  effect  but  other  factors  also  affect  cost  growth  behavior. 

Second,  periods  of  structural  changes  contributed  additional  constant  shifts  in  the  growth.  Two 
such  overlapping  periods  saw  reductions.  One  period  runs  from  1990  to  the  end  of  the  data, 
coinciding  with  the  era  after  the  Goldwater-Nichols  Act  and  other  legislation  that  made  major 
structural  changes  to  the  defense  acquisition  system.  A  second  period  runs  from  2012  to  the  end 
of  our  data,  coinciding  with  the  implementation  of  the  Better  Buying  Power  (BBP)  initiatives  to 
emphasize  cost  consciousness,  improve  efficiency,  strengthen  acquisition  workforce  tradecraft, 
and  seek  ways  to  drive  costs  downward.  These  structural  changes  had  fairly  large  effects:  about  a 
1  percentage-point  reduction  in  the  Goldwater-Nichols  era  and  almost  a  2  percentage-point 
reduction  in  the  overlapping  BBP  era.  Interestingly,  analysis  showed  that  the  apparent  rise  during 
the  acquisition  reforms  of  the  mid-1990s  seen  in  Figure  1  was  correlated  with  budgetary  changes 
rather  than  a  structural  increase  in  growth  in  these  data. 

Third,  these  data  exhibited  a  constant  base  growth  of  just  under  5  percentage  points  (all  other 
things  being  equal). 
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Finally,  our  modeling  indicates  that  the  defense  system  exhibits  a  self-correcting  behavior  that 
adjusts  for  prior  differences  between  the  actual  growth  and  the  anticipated  cost  growth  from 
these  budgetary,  structural,  and  constant  factors.  Such  a  correction  function  adjusts  for 
unforeseen  external  effects  on  the  defense  systems  (including  unanticipated  changes  in  budgets) 
and  internal  variations  (such  as  growth  that  is  different  than  modeled  in  a  particular  year).  This 
correction  function  serves  as  a  kind  of  negative  feedback  that  keeps  costs  under  control. 

Figure  1 :  Actual  Growth  of  Contract  Costs 
Annual  growth  of  contract  costs 

(5-year  moving  average;  adjusted  for  inflation; 


NOTE:  5-year  moving  average  of  annual  growth  in  contracted  total  costs  relative  to  negotiated  cost  targets  on  major  contracts  of  Major  Defense 
Acquisition  Programs  (MDAPs),  as  well  as  major  automated  information  systems  (MAIS)  that  are  MDAPs,  in  Engineering  and  Manufacturing 
Development  (EMD)  and  early  production  that  reported  earned-value  (EV)  data  (i.e.,  almost  no  firm-fixed  price  or  full-production  contracts). 
This  is  different  than  statutory  measures  of  program  cost  growth  relative  to  baselines.  These  data  summarize  18,470  EV  reports  on  1,123  major 
contracts  for  239  MDAPs. 


Figure  2:  Statistical  Model  Fit  to  Actual  Growth  of  EV  contract  costs 
Annual  growth  of  contract  costs 

(5-year  moving  average;  adjusted  for  inflation; 
development  and  early  production) 
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In  addition  to  identifying  what  does  correlate  with  growth  of  EV  contract  costs,  it  is  useful  to  say 
what  does  not.  As  we  made  clear  above,  none  of  this  work  relates  to  government  costs  or  firm- 
fixed-price  costs,  which  are  the  majority  of  production  efforts.  The  two  significant  drops  are  not 
due  to  increased  amounts  of  lower-growth  early  production  contracts  in  those  periods.  In  fact, 
early  production  levels  have  been  relatively  flat  during  the  BBP  era  and  dropping  from  1990- 
2007.  The  reductions  in  the  BBP  period  are  mainly  due  to  reductions  in  growth  on  development 
contracts. 

They  are  also  not  due  to  schedule  effects,  since  schedule  growth  has  been  flat  during  the  BBP  era 
and  does  not  generally  correlate  overall  with  cost  growth  in  these  data. 

Additionally,  they  are  not  due  to  quantity  changes  because  other  analysis  of  contract  data  since 
2000  showed  no  quantity  changes  on  these  kinds  of  contracts  (i.e.,  quantity  changes  are 
predominantly  on  full-rate  production  contracts,  which  are  not  in  these  data). 

Furthermore,  prior  research  tested  a  number  of  other  variables  for  correlation  with  cost  growth 
on  these  contracts  since  the  year  2000,  eliminating  a  large  number  of  them. 1 

In  conclusion,  this  analysis  provides  insights  into  the  behavior  of  the  defense  acquisition  system. 

•  It  behaves  rationally.  It  adjusts  to  budgets  while  reflecting  that  acquiring  systems  that 
have  never  been  built  before  involves  some  cost  risks. 

•  It  responds  to  major  structural  changes.  Recent  efforts  to  strengthen  the  defense 
department’s  cost-conscious  culture  have  improved  cost  control.  Added  oversight  from 
the  era  of  Goldwater-Nichols  and  contemporary  changes  also  appears  to  have  had  a  major 
effect  at  moderating  contract  cost  behavior.  However,  the  reforms  of  the  mid-1990s  did 
not  have  a  large  enough  effect  in  these  data  to  be  measurable. 

•  It  is  stable.  A  feedback  function  self-corrects  for  prior  differences  from  expectations  and 
unforeseen  external  shocks. 

Taken  together,  this  analysis  provides  both  an  encouraging  and  cautionary  tale  that  some  reforms 
can  moderate  costs  in  this  metric  while  others  have  not.  This  analysis  also  demonstrates  how 
statistical  analysis  combined  with  theoretical  insights  deepens  our  understanding  of  the  effects  of 
policy  changes  and  thus  can  inform  future  decision  making  without  resorting  to  supposition. 


1  The  following  variables  were  either  spurious  or  statistically  insignificant:  undefinitized  contract  actions  (although 
UCAs  can  be  significant  in  some  commodity  classes,  especially  ships  in  development);  contract  spending  share  of 
program  spending;  cost-over-target;  share  of  cost  growth  due  to  work-content  growth;  share  of  cost  growth  due  to 
cost-over-target;  margin;  change  in  margin  over  the  contract’s  period  of  performance;  contract  size  (total  dollars); 
schedule  growth;  military  service  (i.e..  Army,  Navy,  Air  Force,  or  DoD);  commodity  type  (except  space  systems  in 
development  and  aircraft  in  early  production);  and  quantity  changes  (see  AT&L,  2015,  pp.  75-77). 
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Introduction 


Are  there  periods  of  improved  cost  control  on  defense  contracts?  Have  recent  efforts  produced 
measurable  results?  What  effects  have  external  events  and  major  policy  changes  had  on 
controlling  costs  on  major  defense  programs? 

To  help  answer  these  questions,  we  examined  the  5-year  moving  average  of  annual  growth  in 
total  EV  contract  costs  on  major  contracts2  for  the  development  and  early  production  of  Major 
Defense  Acquisition  Programs  (MDAPs).  Growth  is  in  real  terms  (i.e.,  after  adjusting  for 
inflation).  This  different  measure  of  growth  reflects  changes  relative  to  negotiated  costs  targets 
on  contracts  tracked  in  earned-value  (EV)  reports.  It  includes  both  scope  growth  (i.e.,  work 
added  to  a  contract  after  award)  and  overruns  (i.e.,  latest  estimate  of  cost  over  the  latest 
negotiated  target).  Negotiated  targets  do  not  necessarily  equal  government  or  contractor 
negotiation  cost  estimates.  These  contract  level  targets  are  also  different  than  broader  program- 
level  cost  estimates,  baselines,  and  statutory  measures  of  cost  growth  relative  to  program-level 
baselines  because  it  excludes  non-contracted  costs  and  the  majority  of  production  contracts. 
Nonetheless,  we  expect  that  the  cost-reimbursement  and  fixed-price  incentive  contracts  that  this 
study  examines  should  show  the  same  trends  as  other  measures  of  total  program  cost  growth — in 
part  because  production  contracts  have  much  lower  cost  growth  than  these  development  and 
early  production  contracts.  EV  reports  are  typically  limited  to  development  (engineering  and 
manufacturing  phase)  and  early  production  contracts.  They  include  fixed-price  incentive  and 
cost-reimbursement  contracts  but  generally  not  firm-fixed  price  contracts  (which  often  are  not 
appropriate  until  full-rate  production  and  usually  do  not  report  EV).  Each  year’s  growth  is  the 
sum  of  all  the  changes  in  active  contract  values  divided  by  the  sum  of  all  the  original  target  costs. 
Thus,  the  portfolio  percentage  change  will  be  closer  in  value  to  the  percentage  change  of  the 
larger  contracts.  Of  final  note,  reports  of  scope  changes  and  estimates  of  final  cost  in  EV  reports 
can  be  sporadic,  leading  to  wide  year-to-year  changes;  therefore,  we  used  a  5-year  moving 
average  to  smooth  out  these  effects  and  reveal  longer-term  trends. 

We  used  standard  statistical  modeling  techniques  to  identify  statistically  significant  factors  that 
are  likely  causes  of  growth.  For  example,  are  reductions  merely  reflections  of  budgetary 
reductions,  or  are  there  measurable  structural  shifts  from  major  policy  changes?  Is  the  defense 
system  stable  in  dealing  with  external  and  internal  spikes  and  shocks?  Of  course,  proving 
causation  is  difficult,  but  correlation  and  coincidence  combined  with  insight  into  defense 
acquisition  can  provide  valuable  insights  into  causes. 


2  Major  contracts  include  the  six  largest  contracts  (prime,  associated,  or  for  government-furnished  equipment)  for 
each  MDAP  valued  at  more  than  $40  million.  Earned  value  data  also  are  available  for  other  MDAP  contracts  of  at 
least  $60  million  in  RDT&E  or  $250  million  in  procurement  or  ship  construction  (in  fiscal  year  [FY]  1990  constant 
dollars).  Note  that  EV  data  are  usually  not  provided  for  firm-fixed  price  (FFP)  contracts,  so  this  MDAP  contract 
dataset  has  very  few  FFPs. 
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Annual  Growth  of  Development  and  Early  Production  Contracted  Costs 

Figure  3  plots  the  5-year  moving  average  of  annual  growth  in  EV  contracted  costs  on  major 
contracts  for  MDAPs.  It  shows  that  some  policy  changes  appear  to  have  resulted  in  dramatic 
downturns  in  the  growth  of  costs  on  executing  contracts  for  these  programs. 

The  Goldwater-Nichols  Act  of  1986 — which  reorganized  the  Department  of  Defense  (DoD)  and 
the  contemporaneous  legislation  that  strengthened  oversight  by  the  Office  of  the  Secretary  of 
Defense  on  defense  acquisition  programs — coincided  with  a  major  downturn  in  annual  growth  of 
EV  contract  costs  at  point  #2  on  the  chart.  Also,  the  initiation  of  the  recent  DoD  initiatives  called 
Better  Buying  Power  (BBP)  to  instill  cost  consciousness,  improve  efficiency,  strengthen  the 
workforce,  and  seek  ways  to  drive  costs  downward  also  coincided  with  the  recent  downturn 
starting  at  point  #4.  At  3.5%  in  FY2015  for  the  portfolio,  the  department  is  now  at  the  lowest 
level  of  growth  since  before  FY1985. 

Conversely,  policy  changes  starting  in  the  mid-1990s  that  reduced  oversight  on  contractor 
performance  appear  to  coincide  with  an  upturn  in  annual  added  costs  on  contracts  from  point  #3. 
We  test  below  whether  that  period  increase  is  statistically  significant  (e.g.,  whether  the  trend  is 
really  flat  from  1995  if  you  dismiss  1996  as  anomalous). 

Increased  demands  for  new  capabilities  can  also  result  in  costs  added  to  contracts;  that  is  visible 
in  the  chart  during  the  defense  buildup  in  the  1980s  by  President  Reagan  and  during  the  post- 
9/1  1  wars  to  combat  global  terrorism. 


Figure  3:  5-year  Moving  Average  of  Annual  Growth  of  EV  contract  costs  (FY1 985-201 5) 


NOTES:  The  5-year  moving  average  of  annual  growth  in  contracted  total  costs  are  relative  to  negotiated  cost  targets 
on  major  contracts  of  MDAPs  (including  MAIS  that  are  large  enough  to  also  be  MDAPs)  in  engineering  and 
manufacturing  development  (EMD)  and  early  production  that  reported  earned-value  (EV)  data  (i.e.,  include  almost  no 
firm-fixed  price  or  full-production  contracts).  This  is  different  than  statutory  measures  of  program  cost  growth  relative 
to  Milestone  B  baselines.  These  data  summarize  1 8,470  earned-value  reports  on  1 ,1 23  major  contracts  for  239 
MDAPs.  TSPR  =  Total  System  Performance  Responsibility. 
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Factors  Contributing  to  Annual  Growth  in  EV  contract  costs 


To  understand  whether  the  trends  in  Figure  3  are  significant  and  what  may  be  driving  them,  we 
employed  the  following  statistical  analyses.  First,  we  wanted  to  determine  whether  the  two  steep 
drops  are  statistically  significant  and  whether  they  are  merely  an  artifact  of  budget  cuts.  We  also 
wanted  to  test  whether  the  growth  in  the  mid-to-late  1990s  was  flat  or  significantly  upward. 

We  found  that  the  combined  effects  of  three  types  of  drivers — together  with  a  constant  base  and 
random  noise — very  closely  fit  the  dynamics  of  the  growth  curve. 

1.  Budget  effects.  The  growth  in  costs  is,  in  part,  positively  correlated  with  budget  levels 
and  dynamics.  This  includes  two  factors: 

-  the  average  of  the  DoD  budget  over  the  past  5  years,  and 

-  the  change  in  that  average  from  the  prior  year  to  the  current  year. 

2.  Two  structural  changes.  Two  overlapping  periods  correlate  with  partial  reductions: 

-  one  since  1990,  coincident  with  post-Goldwater-Nichols  implementation,  and 

-  one  since  2012,  coincident  with  BBP  implementation.3 

3.  A  self-correcting  behavior.  A  factor  that  corrects  for  prior  differences  between  the 
anticipated  cost  growth  from  the  budgetary,  structural,  and  constant  factors  and  the  actual 
growth.  This  autoregression  factor  adjusts  for  unforeseen  external  “shocks”  to  the 
systems  (including  unanticipated  changes  in  budgets)  and  internal  variations  (such  as 
larger-than-modeled  annual  contract  obligations).  It  serves  as  a  kind  of  negative  feedback 
that  keeps  the  system  under  control. 

Of  note,  we  also  tested  for  structural  change  during  the  period  of  the  mid-to-late  1990s 
(FY 1995-2001)  in  the  era  of  Reinventing  Government.  The  effects  in  that  era  failed  the 
statistical  tests  in  these  data  (i.e.,  we  cannot  claim  that  they  are  statistically  different  from  zero). 
Figure  4  shows  that  the  growth  during  this  period  appears  relatively  flat  after  subtracting  the 
partial  budgetary  effects. 

Figure  4:  Actual  Growth  Less  Modeled  Budget  Effects  (FY1 985-201 5) 


Total  actual  growth  of  contracted  costs  beyond  budget  model  effects 
(5-year  moving  average) 


3  Since  the  effects  of  new  policies  take  time  to  show,  we  measured  these  periods  to  start  2-3  years  after  their 
enactment.  Goldwater-Nichols  Act  was  signed  into  law  at  the  start  of  FY1987,  and  the  first  BBP  memoranda  were 
issued  in  late  FY2010. 
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Testing  for  Underlying  Causes 

What  may  be  the  underlying  causes  for  the  factors  identified  above?  While  it  is  difficult  to  trace 
changes  to  individual  policy  changes,  we  can  identify  some  underlying  drivers  and  rule  out 
others. 

Development  and  Early  Production  Differences 

BBP-era  drops  are  driven  by  dropping  development  cost-growth  rates,  not  higher  proportions  of 
early  production  contracts.  The  recent  downward  trend  is  not  driven  by  an  increased  fraction  of 
early  production  dollars  in  the  annual  portfolios.  Because  growth  on  early  production  contracts 
has  been  lower  than  that  for  development  since  1990,  such  a  shift  might  have  been  a  logical 
cause  for  the  recent  drop.  In  fact,  however,  the  proportions  between  development  and  early 
production  dollars  have  been  relatively  flat  since  2012  (see  Figure  5).  Instead,  the  overall  BBP- 
era  drop  is  being  driven  by  steep  drops  in  the  growth  rates  on  development  contracts  (see  Figure 
6). 

Initial  Goldwater-Nichols-era  drops  are  largely  driven  by  dropping  early  production  cost  rates, 
not  higher  proportions  of  early  production  contracts.  Similarly,  the  dramatic  initial  drop  from 
FY1987-1991  as  the  Goldwater-Nichols  era  changes  were  first  being  implemented  was  driven 
mostly  by  steep  drops  in  growth  rates  on  early  production  contracts  (see  Figure  6).  The 
proportion  of  early  production  dollars  in  the  portfolio  was  mostly  flat  in  this  early  period.  After 
this  initial  period,  both  the  development  cost  rate  and  proportions  were  increasing,  so  the  overall 
Goldwater-Nichols  era  structural  effect  is  not  due  to  the  development/production  split  or  their 
cost  dynamics. 

Schedule  Effects 

DoD  genemlly  was  not  adjusting  schedule  to  lower  costs.  We  were  also  able  to  reject  the 
possibility  that  the  DoD  was  simply  adjusting  contract  schedules  to  lower  growth  of  EV  contract 
costs  in  the  BBP  era  or  overall.  Overall,  we  found  no  correlation  between  growth  of  costs  and 
schedule  on  these  contracts.  The  5-year  moving  averages  of  growth  in  annual  schedules  and  EV 
contract  costs  are  statistically  independent.4  We  can  see  this  in  Figure  7,  where  cost  and  schedule 
growth  were  moving  in  opposite  directions  since  about  2002. 

Quantity  Effects 

These  factors  are  independent  of  quantity  changes.  Lastly,  we  note  that  prior  research  showed 
that  contract  quantity  almost  never  changes  on  these  early  production  contracts — at  least  since 
about  the  year  2000  (see  AT&L,  2015).  Also,  quantity  generally  is  not  an  element  on 
development  contracts  (except  for  relatively  few  test  articles).  Therefore,  quantity  is  not  an 
underlying  cause  for  the  factors  in  the  statistical  model  or  for  the  separate  growth  dynamics  by 
development  and  early  production. 


4  i.e.,  they  failed  the  Spearman  correlation  test. 
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Figure  5:  Portfolio  Split  between  Development  and  Early  Production  Contracts 


Fraction  of  portfolio  base 
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NOTE:  This  reflects  the  relative  total  dollars  in  the  contract  bases  each  year. 


Figure  6:  Separate  Growth  Rates  for  Development  and  Early  Production  Contracts 


Annual  Growth  of  Contracted  Costs 

(5-year  moving  average) 


FY 
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Figure  7:  Comparing  Growth  in  Schedule  and  Cost  on  Major  Contracts  (FY1 985-201 5) 


5-Year  Moving  Average  of  Annual  Growth 


FY 

NOTE:  In  the  BBP  era  (since  2012),  schedule  growth  is  essentially  flat  while  cost  growth  has  dropped  dramatically. 


Cycle  Time,  Scope  Growth,  and  Commodity  Effects 

Cycle  time,  scope  growth,  and  certain  commodities  correlate  with  growth  of  costs  in  these  data. 
Prior  research  did  find  three  variables  that  generally  correlate  with  growth  in  this  type  of  contract 
data:  the  original  length  of  the  contract  (i.e.,  cycle  time,  not  schedule  growth),  scope  growth,  and 
two  commodities  (space  systems  in  development  and  aircraft  contracts  in  early  production). 
Cycle  time  was  the  largest  correlate  (AT&L,  2015,  pp.  75-77). 

Should-Cost  Reductions 

Reduced  cost  growth  in  the  BBP  era  may  be  due  in  part  to  Should-Cost  savings.  Finally,  we  note 
that  the  DoD  has  continued  to  see  increasing  savings  on  programs  from  Should-Cost  initiatives 
in  BBP.  These  savings  have  grown  across  the  acquisition  portfolio,  so  these  may  have  additional 
effects  in  lowering  scope  growth  and  cycle  time  at  the  contract  level. 

Other  Variables 

Prior  research  tested  a  number  of  other  variables  that  do  not  correlate  with  growth  of  costs  on 
these  contracts.  Further,  prior  research  tested  a  number  of  other  variables  for  correlation  with 
price  and  cost  growth  on  these  contracts  since  the  year  2000.  It  found  the  following  variables 
were  either  spurious  or  statistically  insignificant:  undefinitized  contract  actions  (although  UCAs 
can  be  significant  in  some  commodity  classes,  especially  ships  in  development),  contract 
spending  share  of  program  spending;  cost-over-target;  share  of  cost  growth  due  to  work-content 
growth;  share  of  cost  growth  due  to  cost-over-target;  margin;  change  in  margin  over  the 
contract’s  period  of  performance;  contract  size  (total  dollars);  schedule  growth;  military 
department  (i.e.,  Army,  Navy,  Air  Force,  or  DoD);  commodity  type  (except  space  systems  in 
development  and  aircraft  in  early  production);  and  quantity  changes  (see  AT&L,  2015,  pp.  75- 
77). 
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Conclusions 


This  analysis  provides  some  insights  into  the  behavior  of  the  defense  acquisition  system. 

•  It  is  stable.  The  autoregression  feedback  provides  a  self-correcting  control  system. 

•  It  behaves  rationally.  It  adjusts  to  budgetary  changes,  prior  differences  from  expectations, 
and  unforeseen  external  shocks. 

•  It  responds  to  major  structural  changes.  Added  oversight  from  Goldwater-Nichols  Era 
changes  appears  to  have  had  a  major  effect  at  moderating  contract  cost  behavior. 
Similarly,  recent  efforts  to  instill  a  cost-conscious  culture  have  improved  cost  control. 

The  reforms  of  the  mid-1990s  were  coincident  with  an  undesirable  trend,  though  unlike 
the  positive  trends,  our  modeling  did  not  show  statistically  significant  indications  of 
causal  factors. 

Taken  together,  this  analysis  provides  both  an  encouraging  and  cautionary  tale  that  some  reforms 
can  moderate  costs  in  this  metric  while  others  have  not.  This  analysis  also  demonstrates  how 
statistical  analysis  combined  with  theoretical  insights  provides  useful  insights  into  the  effects  of 
policy  changes  and  thus  can  inform  future  decision  making  without  resorting  to  supposition. 
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Analytic  Details 


Data 

We  analyzed  growth  in  contract  cost  using  summary  EV  data  on  1,123  major  contracts  from  FY1981-2015  for  239  major  defense 
acquisition  programs  (MDAPs).  These  included  the  combined  results  from  9,680  EMD  reports  and  8,790  early  production 
reports.  Table  1  lists  the  actual  5-year  moving  average  of  annual  growth  of  EV  contract  costs  as  calculated  using  the  equations 
below. 


Table  1.  5-Year  Moving  Average  of  Annual  Growth  of  EV  contract  costs  (FY1985-2015) 


FY 

Growth  (actual) 

FY 

Growth  (actual) 

1985 

5.47% 

2001 

6.67% 

1986 

8.89% 

2002 

6.56% 

1987 

9.84% 

2003 

6.82% 

1988 

9.33% 

2004 

7.06% 

1989 

8.00% 

2005 

8.55% 

1990 

8.07% 

2006 

7.98% 

1991 

5.99% 

2007 

7.07% 

1992 

5.65% 

2008 

8.66% 

1993 

6.02% 

2009 

8.16% 

1994 

6.07% 

2010 

7.48% 

1995 

5.93% 

2011 

9.10% 

1996 

5.07% 

2012 

8.21% 

1997 

6.20% 

2013 

6.05% 

1998 

6.14% 

2014 

5.27% 

1999 

2000 

6.75% 

6.12% 

2015 

3.49% 

NOTE:  Results  reflect  18,470  earned-value  reports  on  1,123  major  contracts  for  239  major  defense  programs. 


These  growths  are  calculated  across  all  contracts  together  by  totaling  the  changes  of  the  program  manager’s  estimate  at 
completion  (PM  EAC)  in  the  year  (i.e.,  from  the  last  report  from  the  prior  year)  for  all  contracts,  then  dividing  by  the  sum  of  all 
the  initial  contract  cost  targets  for  all  the  active  contracts  in  a  year: 

E^=1A  PMEACr{f) 

9  Yg=1CBB  ^original) 

for  m  reports  on  n  contracts  active  in  year  t 


where: 

g  (t)  is  the  average  growth  of  contracted  costs  across  all  active  EV  contracts  in  year  t  (adjusted  for  inflation), 

APMEACr{t)  is  the  change  in  contract  cost  as  reported  by  the  program  manager’s  estimate  at  completion  (PMEAC)  for  EV 
report  r  for  an  active  program  in  year  t,  after  adjusting  for  inflation, 

CBB^originaV)  is  the  original  negotiated  cost  target5  at  contract  award  for  contract  i  in  common  base-year  dollars,  and 
t  is  the  FY  for  the  annual  growth. 

Given  g^,  the  5-year  moving  average  growth  Gf  of  EV  contract  costs  for  year  t  is  simply 

^  _Sf=o^(t-0 
G‘-  5 

fort  £  [FT1985.FY2015]. 


5  In  EV  reports,  the  Contract  Budget  Base  (CBB)  is  the  sum  of  the  Negotiated  Contract  Cost  (NCC)  and  the  Authorized  Unpriced 
Work  (AUW). 
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Model  of  Annual  Growth  of  EV  contract  costs 


The  following  equations  show  our  statistical  model  of  the  5-year  moving  average  of  annual  growth  of  EV  contract  costs  on  major 
MDAP  contracts: 


where 


Gt  ~  co  +  ciBt-i  +  c2 +  c3GNt  +  c4BBPt  +  ut 

Ut  =  C5Ut_  5  +  £ 

£  ~  i.  i.  d.  N(0,  er2) 

for  f  £  [TY1985,TY2015]. 


Gt  is  the  5-year  moving  average  of  annual  growth  of  EV  contract  costs  at  time  t, 

Ci  are  coefficient  constants, 

Bt_1  is  the  5-year  average  of  the  DoD  budgets  from  time  t-5  to  time  t-1, 

A Bt  =  Bt  —  Bt_a  is  the  change  in  the  5-year  moving  average  of  the  budget  from  time  t-1  to  time  t. 


GNt  = 
BBPt 


l  fort  >FY  1990, 

0  for  t  <  FY 1990; 

fl  fort  >  FY2012, 
tO  fort<  FY2012; 


this  is  the  indicator  variable  for  a  hypothesized  structural  change  since  FY1990 
(i.e.,  the  era  of  full  implementation  of  the  Goldwater-Nichols  Act), 

this  is  the  indicator  variable  for  a  hypothesized  structural  change  since  FY2012 
(i.e.,  the  era  of  full  implementation  of  BBP), 


ut  is  an  autoregressive  factor  for  the  difference  between  the  actual  growth  and  the  growth  predicted  by  the  other  factors 
5  years  earlier,  and 

£  is  a  series  of  independent  and  identically  distributed  ( i.i.d .)  samples  from  a  normal  distribution  with  zero  mean 
(i.e.,  Gaussian  white  noise). 


Table  2  lists  the  coefficients  c,  for  the  statistically  significant  factors  in  the  model  of  the  5-year  moving  average  of  annual  growth 
of  EV  contract  costs  on  major  MDAP  contracts  reporting  EV.  Figure  8  shows  the  partial  contributions  of  these  drivers  to  the 
growth  curve  for  each  year  from  FY1985-2015,  and  Figure  9  shows  the  combined  effect  of  the  two  budgetary  variables.  Recall 
that  Figure  2  shows  how  closely  this  statistical  model  matches  the  actual  growth  in  EV  contract  costs  over  this  period;  it  uses  the 
MLE  coefficients. 


Table  2.  Model  Coefficients  and  Statistics:  5-Year  Moving  Average  of  Annual  Growth  of  EV  contract  costs 


Driver  Type 

Factor 

OLS 

(Newey  West) 

MLE 

(OIM) 

MLE 

(OPG) 

p-value  OLS 

p-value  MLE 
(OIM) 

p-value  MLE 
(OPG) 

Budget 

Effects 

Ci  coefficient  for  each  $100B  in  the  5-year 
moving  average  of  prior  DoD  TOAs  from  last 
year  (standard  error) 

0.602% 

(0.0000195) 

0.551% 

(0.0000197) 

0.551% 

(0.0000319) 

0.005*** 

0.005*** 

0.084* 

C2  coefficient  for  each  $10B  change  in  5-year 
moving  average  of  prior  DoD  TOAs  from  last 
year 

(standard  error) 

0.233% 

(0.0000788) 

0.252% 

(0.0000587) 

0.252% 

(0.0000729) 

0.007*** 

0.000*** 

0.001*** 

Structural 

Effects 

Cj  coefficient  for  Goldwater-Nichols  era 
structural  change,  FY1990-2015 
(standard  error) 

-1.07% 

(0.00526) 

-0.948% 

(0.00457) 

-0.948% 

(0.00434) 

0.051* 

0.038** 

0.029** 

C4  coefficient  for  BBP  era  structural  change, 
FY2012— 2015 
(standard  error) 

-1.70% 

(0.00851) 

-1.87% 

(0.00776) 

-1.87% 

(0.00961) 

0.057* 

0.016** 

0.051* 

Self- 

Corrections 

Cs  coefficient  autocorrelation  coefficient: 
correction  amount  of  actual-to-model 
difference  in  prior  year 
(standard  error) 

-0.334 

(0.157) 

-0.583 

(0.208) 

-0.583 

(0.231) 

0.044** 

0.005*** 

0.012** 

Base 

Co  coefficient  for  constant 
(standard  error) 

4.63% 

(0.0122) 

4.80% 

(0.0120) 

4.80% 

(0.0176) 

0.001*** 

0.000*** 

0.006*** 

*  statistically  significant  at  the  10%  level  of  significance.  ***  statistically  significant  at  the  1%  level  of  significance. 

**  statistically  significant  at  the  5%  level  of  significance 

NOTES:  These  results  are  robust  to  the  different  estimation  methods  shown  and  to  different  ways  of  calculating  the  variance  covariance  matrix. 
The  autocorrelation  coefficients  met  stability  conditions.  After  accounting  for  autocorrelations,  the  residual  error  was  white  noise. 
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Figure  8:  Factors  Contributing  to  5-year  Moving  Average  of  Annual  Growth  of  EV  contract  costs  (FY1985-2015) 


Partial  Contributions  (MLE  model) 


Base  constant  (overall) 

4.8  %-points 

Average  budget  over  prior  5  years 

+  0.55  %-points  per  $100B 

Change  in  5-year  average  of  past  budgets  from  the  prior  year  to  current  year 

+  0.25  %-points  per  $10B  change 

Post-Goldwater  Nichols  era  structural  change 
(since  FY1990) 

-  0.95  %-points 

BBP  era  structural  change 
(since  FY2012) 

-  1.9  %-points 

Self-correction  factor  of  difference  between  actuals  and  the  other  factors  5 

years  ago 

-  0.58  times  the  overage  5  years  ago 

Randomness 

+  pure  noise 

Figure  9:  Factors  Contributing  to  5-year  Moving  Average  of  Annual  Growth  of  EV  contract  costs:  Combined  Budgetary 

Factors  (FY1 985-201 5) 
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In  simple  terms,  this  model  (not  surprisingly)  indicates  that  the  defense  acquisition  system  modulates  changes  to  major  program 
contracts — in  part — based  on  budgets:  if  there  are  more  resources  then  the  system  addresses  more  issues  such  as  threats  and 
engineering  issues  that  arise  in  development  and  early  production.  When  resources  are  tight,  the  system  addresses  fewer  of  these 
problems.  The  combined  partial  effect  is  that  growth  changes  somewhat  in  the  same  direction  that  budgets  change  (i.e.,  are 
procyclical),  which  can  be  seen  visually  in  Figure  9.  Cost  change  is  expected  to  increase  just  over  half  a  percentage  point  for 
every  $100  billion  in  average  budget  over  the  prior  5  years.  The  model  also  predicts  that  cost  changes  will  increase  about  a 
quarter  of  a  percentage  point  for  every  $10  billion  change  in  the  5 -year  moving  average  of  past  budgets  from  last  year  to  the 
current  year. 

Also,  there  have  been  additional  behavioral  shifts  to  partially  reduce  growth  on  major  contracts:  one  coincident  with  the 
implementation  of  Goldwater-Nichols  to  present,  and  one  coincident  with  BBP  implementation  to  present.  These  effects  are 
large,  statistically  significant,  and  independent  of  budgetary  dynamics.  In  the  Goldwater-Nichols  era,  growth  has  been 
systemically  moderated  by  about  1  percentage  point,  and  growth  has  been  reduced  almost  another  2  percentage  points  in  the  BBP 
era  (all  other  things  being  equal).  Measured  against  the  peak  annual  growth  of  almost  10  percent  in  this  period,  these  reductions 
constitute  a  systemic  reduction  of  almost  a  third. 

In  addition,  the  department  self-corrects  for  external  and  internal  variations  that  ensures  stability.  If  contract  work  grew  in  a  year 
more  than  what  we  would  expect  from  the  model  given  budgetary  trends,  existing  structural  behaviors  (say,  because  of  critical 
threats),  and  prior  corrections,  then  the  DoD  cuts  back  in  the  future.  Conversely,  if  contract  work  grew  less  than  expected,  then 
the  department  tends  to  increase  obligations  in  the  following  year  to  address  problems.  In  other  words,  it  shows  that  the  defense 
acquisition  system  self-corrects  for  differences  between  what  it  anticipates  it  can  obligate  on  contracts  given  recent  budgetary 
effects  under  the  then-current  structural  climates  and  what  it  actually  put  on  existing  contracts.  In  addition  to  external  and  random 
shocks,  this  factor  may  reflect  a  so-called  “horsetail”  effect  wherein  the  DoD’s  5-year  future-years  defense  program  (FYDP)  plan 
usually  fails  to  predict  both  downward  and  upward  cyclic  changes  in  actual  budgets  (i.e.,  being  overly  optimistic  when  budgets 
are  declining  and  pessimistic  when  budgets  are  rising — see,  for  example,  Lambert,  2014,  p.  3,  and  Harrison,  2014,  p.  21). 

Lastly,  there  is  a  constant  base  of  growth  on  contracts  over  all  these  years,  reflecting  that  defense  weapon  system  development 
and  early  production  involves  some  remaining  uncertainties,  risks,  and  investments  (including  changes  to  keep  up  with  evolving 
threats).  The  overall  base  of  just  under  5  percent  annually,  however,  is  relatively  low  given  these  are  the  more  risky  development 
and  early  production  contracts,  not  full-rate  production. 


Model  of  Annual  Contract  Schedule  Growth 

Statistically  we  found  that  schedule  growth  has  been  on  a  simple  downward  trend  along  with  its  own  one-year  self-correction 
factor  (see  Table  3). 

Below  are  the  statistically  significant  factors  in  the  model  of  the  5-year  moving  average  of  annual  growth  of  contract  schedules 
on  major  MDAP  contracts  reporting  EV. 


St  =  C0+  ClYt  +  Ut 
Ut  =  C2  Ut_!  +  £ 

£  —  i.  i.  d.  N(0,  a2) 

fort  G  [FY1985,FY2015]. 

where 

st  is  the  5-year  moving  average  of  annual  schedule  growth  for  active  major  MDAP  contracts  in  year  t, 

Ci  are  coefficient  constants, 

Yt  is  the  variable  of  time  (i.e.,  the  model  is  a  secular  time  trend), 

ut  is  an  autoregressive  factor  for  the  difference  in  the  prior  year  between  the  actual  schedule  growth  and  the  growth 
predicted  by  the  other  factors,  and 

£  is  a  series  of  independent  and  identically  distributed  ( i.i.d .)  samples  from  a  normal  distribution  with  zero  mean 
(i.e.,  Gaussian  white  noise). 

Table  3  lists  the  coefficients  c,-  for  the  statistically  significant  factors  in  the  model  of  the  5-year  moving  average  of  annual 
schedule  growth  on  major  MDAP  contracts  reporting  EV. 
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Table  3.  Model  Coefficients  and  Statistics:  5-Year  Moving  Average  of  Annual  Growth  of  Contract  Schedules 


Driver  Type 

Factor 

MLE 

p-value 

(OPG) 

MLE 

(OPG) 

Linear  declining 

Coefficient  for  time  trend  (per  year) 

-0.157% 

0.014** 

trend 

(standard  error) 

(0.000651) 

Self-Corrections 

Autocorrelation  coefficient:  correction  amount 
of  actual-to-model  difference  in  prior  year 
(standard  error) 

0.766 

(0.154) 

0.000*** 

Base 

Coefficient  for  constant 

3.21% 

0.016** 

(standard  error) 

(1.30) 

**  statistically  significant  at  the  5%  level  of  significance 
***  statistically  significant  at  the  1%  level  of  significance 

NOTES:  Annual  5-year  moving  average  of  schedule  growth  for  executing  contracts  for  major  programs  has  a  negative  secular  time  trend  over 
the  last  31  years  (i.e.,  schedule  growth  is  declining).  Each  year's  growth  is  the  sum  of  all  the  changes  in  active  contract  schedules  divided  by  the 
sum  of  all  the  original  schedules.  Thus,  the  portfolio  percentage  change  will  be  closer  in  value  to  the  percentage  change  of  the  longer  contracts 
(similarly  to  how  the  growth  in  EV  contract  costs  is  affected  by  larger  programs).  The  autocorrelation  coefficient  met  stability  conditions.  After 
accounting  for  autocorrelation,  the  residual  error  was  white  noise. 


Statistical  Tests  Employed 

For  estimation  methods  we  employed  Ordinary  Least  Squares  (OLS)  and  Maximum  Likelihood  Estimation  (MLE).  For 
calculating  the  correct  variance-covariance  matrix  we  employed  three  different  methods:  Newey-West,  Observed  Information 
Matrix  (OIM),  and  Sum  of  outer  product  of  gradient  vectors  (OPG). 
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